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Background / Context: 

The effect of student mobility on student outcomes has garnered much attention by 
researchers and policy-makers over the past few decades (e.g., Hanushek, Kain, & Rivkin, 2004). 
Out of this work, researchers have found that a specific type of mobility, structural moves when 
students age out of a given school (e.g., k-5 to 6-8), is associated with negative shocks to students 
short- and long-term achievement (Dhuey, 2013; Schwartz, Stiefal, Rubenstein, & Zabel, 2011; 
Schwerdt, & West, 2013; Rockoff, & Lockwood, 2010). However, the extant research relies on 
annual tests, typically measured each spring, to estimate the effect of structural moves on student 
achievement. The current literature does not allow researchers to evaluate whether the negative 
shock occurs at the start of the school year, or develops over time as students adjust to their new 
setting. Using a unique data set that includes both fall and spring test scores, we set out to evaluate 
whether students fall achievement, presumably before instruction occurs, is affected by the 
structural move, and whether students’ school-year achievement growth is affected as well. Our 
preliminary results suggest that students experience a negative shock to achievement at the start of 
the year. Our results have the potential to shed new light on the impact of structural transitions, 
and have important implications for policy and practice. 

Purpose / Objective / Research Question / Focus of Study: 

We estimate the overall impact of middle school transitions on students’ achievement, both 
in terms of test score performance in the fall as students first arrive in their new school, as well as 
their learning rate throughout the subsequent school year(s). We build off summer learning loss 
research which suggests that minority students may be more likely to lose ground during the 
summer and therefore start each school year off behind. We therefore also explore whether 
minority students’ growth trajectories are more severely affected by middle school transitions. The 
current project fits nicely with the conference theme, which specifically focuses on “educational 
trajectories and transitions, from: (a) preschool to kindergarten, (b) elementary to middle school, 
(c) middle to high school, (d) high school to college, and (e) into adulthood.” 

Setting: 

The data for the current study comes from a partnership with the Northwest Evaluation 
Association (NWEA), which administers the Measures of Academic Progress (MAP) assessment 
to school age students across the U.S. The MAP is a computer adaptive test that assesses student 
performance in math and reading. The MAP assessment is not used in accountability policies, but 
rather serves as a formative tool delivered to teachers and schools multiple times per year. NWEA 
reports test scores in an IRT-based metric, which is vertically equated and equal-interval scaled. 

Population / Participants / Subjects: 

The dataset includes students enrolled in public schools who take the MAP assessment. 
The Southern State uses the MAP assessment in all public school districts for the past six years. 
In theory, all students in public schools should take the MAP assessment twice per year (fall and 
spring) in grades two through nine, however for idiosyncratic reasons, many students may not have 
test scores at every testing occasion. Table 1 presents sample sizes by grade and year of data (see 
appendix for tables, figures, and equations). We are particularly focused on the subset of students 
who make a middle-school transition after fourth, fifth, or sixth grade, as well as students who 
attend K-8 schools and therefore make no middle-school transition. 

Intervention / Program / Practice: 
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We examine the impact of structural mobility on student performance. Extant research 
documents that structural transitions are difficult for students, and that student achievement is 
disrupted — either temporarily or permanently — as a result of these moves. The study of structural 
mobility requires robust data that differentiates between a structural move and a student who leaves 
the district. Furthermore, because large-scale testing typically occurs once per year in the spring, 
it is often impossible to detect if the start of the new school year might be particularly difficult. 

For a subset of the students in our Southern State whom we follow across our five-year 
longitudinal panel, we observe their MAP test score trajectory as they move from 4 th through 8 th h 
grade. A structural transition occurs as students complete the final grade level offered at their 
elementary school and are re-assigned to a new middle school. Although the modal transition in 
the state is between 5 th and 6 th grade, a non-trivial share of elementary schools end in 4th or 6 th 
grade. We will use this variation to estimate the effect of structural moves on students’ middle 
school achievement using either students’ own histories as a counter-factual (e.g., student fixed- 
effects) or students who do not make structural moves (e.g., students in K-8 schools) as a counter- 
factual. These different approaches to timing of structural transitions allows us some variability in 
the treatment of interest — structural transitions — that we can use to explore the impact on student 
achievement both in terms of immediate performance (fall scores), as well as future performance 
(following students through the end of middle school). Of course, the variation in when students 
experience their middle school may not be exogenous — a complication we seek to address through 
various quasi-experimental approaches described in the next section in more detail. 

Research Design: 

The treatment of interest is the impact of middle school transitions on immediate (fall) and 
longer term (throughout middle school) student achievement. We start by modeling students’ math 
or reading test scores as a function of time in a multi-level modeling framework. We use a two- 
level random effects (hierarchical) model, where the outcome of interest is a test score, Yijk, 
assigned to student i at grade- seme ster j, in school k. Each student’s vector of test scores (fall and 
spring, across years) are modeled at level one as repeated observations nested within students. At 
level one, students’ growth trajectories are modeled using a set of dummy variables for each grade- 
semester — Gr4_Fall, Gr4_Spr, Gr5_Fall, Gr5_Spr, etc. — that are coded as 1 if the observation 
occurred after the given grade-semester. This coding scheme structures the level-one coefficients 
so that they represent individual students’ grade-specific school-year and summer learning rates. 
One of the main advantages of this approach is the ability to estimate both the mean and variance 
across students in school-year and summer learning rates. (See Equation (1) in the appendix). We 
first use Equation (1) to examine overall mean growth trajectories, as well as to estimate the 
variance across schools. These preliminary descriptive explorations will provide us with a basis 
for thinking about how to best approach modeling student growth trajectories over time (e.g., 
linear, quadratic, logarithmic, etc). 

In order to explore whether variability in middle school transitions are associated with 
decrements to student learning, we change the temporal structure of Equation (1) to best represent 
the overall pattern of student learning based on our initial descriptive exploration (for the sake of 
simplicity, we assume in this proposal that a simple linear model is most appropriate). We then 
introduce a new time-varying indicator that identifies when a structural move at the start of the 
school year occurs, whenever that might be (e.g., start of 5 th , 6 th , or 7 th grade). The inclusion of the 
indicator allows us to test whether an immediate decrement in test scores is detectable in the fall. 
Furthermore, we interact the transition indicator variable with the linear time variable in order to 
explore whether the rate of learning subsequent to the structural transition differs from each 
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student’s pre-transition growth rate. We have also estimated school value-added measures so that 
we can control for the quality of both the school left, as well as the school attended after the 
structural transition. Students who transition to schools of lower relative quality may experience a 
decrement to learning that is not due to the experience of being at a new school, but rather to the 
change in the quality of instruction, and so in some models we account for the relative difference 
in school quality (as measured imperfectly through school value-added measures). While on the 
one hand, it seems straightforward that one would want to control for post-transition school 
quality, there is some possibility that school quality is endogenous to transition timing. 

Because students make structural transitions at different points in time, we can disentangle 
any grade- specific learning patterns from the timing of the structural transition itself. Furthermore, 
we can control for some time-invariant or pre-fourth grade characteristics of students including 
their fourth grade test scores, gender, and race/ethnicity. At the within-student level, we can also 
control for non- structural moves across schools, as well as student grade repetition or skipping. To 
some extent, these time- varying and time-invariant controls ensure that any comparisons in test 
scores when students are making structural moves or not are among more “similar” conditions, 
however we will discuss other approaches to isolate the variation in treatment status, below. 

Our random effect strategy is vulnerable to omitted variable bias due to systematic 
differences between students who do and do not make structural moves. We will also pursue a 
random effects model that removes fixed differences among students while maintaining our ability 
to estimate the variation in the effect of structural moves across students. This approach is akin to 
a “within-student” approach, because we compare each student’s achievement before and after the 
structural move and estimate the coefficient on transitioning only using within-student variation. 

It is important to think about the systemic ways in which students may be sorted into 
structural transitions at different timings and address any non-random sorting in the analytic 
approach. On the one hand, we believe that families are probably unlikely to select which school 
their child attends at the start of formal schooling based on whether they prefer transitions to occur 
after a particular grade level. In addition, for some families, geographic constraints may determine 
whether and when their children experience the middle school transition: For instance, perhaps 
only one particular structural transition timing is available to them because, for example, all 
schools in the given districts transition after fifth grade. In this sense, families are less likely to 
self-select into a given treatment timing. That said, it seems possible that some families could have 
some preferences about middle school transitions, perhaps preferring to send their children to K-8 
schools in which no middle school structural move occurs. 

Ultimately, our most powerful source of evidence on whether middle school transitions 
negatively affect student achievement and learning rates will come from the variability in transition 
timing. If, for instance, we see a decrement to student learning appears at the start of 6 th grade for 
those who transition summer after 5 th grade but no such decrement exists for students who 
transition at a later time, then this suggests the structural transition itself could be the cause. We 
would be further convinced if we also see that students who transition in 7 th grade (who did not 
exhibit a drop in test scores at the start of 6 th grade when they did not transition) in fact do exhibit 
a drop in fall test scores at the start of 7 th grade, while the earlier transitions do not. 

In order to attend to the fact that students may not be randomly assigned to whether they 
experience a structural transition or not (and if so, after what grade), we explore several quasi- 
experimental approaches to identifying some random variation in the treatment of interest. For 
instance, we explore the feasibility of using district fixed effects so that we can hold constant the 
public school settings. We also explore the feasibility of using an instrumental variables approach, 
in which the instrument is the terminal grade of the school the students first attends in kindergarten. 
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Finally, one avenue which we have not yet explored is whether the given state has schools that 
have changed their grade-level structure since 2007-08. For instance, a given elementary school 
might change from serving grades K-4 to serving grade K-5. We could compare the outcomes of 
students in one cohort who made the structural transition after 4 th grade, to students in the 
subsequent cohort who made the transition after 5 th grade. Such an approach could improve our 
ability to identify exogenous variation in transition timing, however the subset of schools that 
make such changes may not be representative of the larger population. 

Data Collection and Analysis: 

We currently have the data and have cleaned and used it for other analyses. Our previous 
work with this data increases the feasibility of the current proposed work. Of the approximately 
170,000 students for whom we observe achievement throughout 4 th through 8 th grade, 
approximately 20 percent do not make a structural transition before the start of sixth grade. This 
suggests that there is variation in exposure to and timing of middle school transitions in our sample. 

Findings / Results: 

We have conducted preliminary analyses following one cohort of students as they move 
through grades 3 through 7. In Figure 1, we present predicted model-based growth trajectories 
estimated using Equation (1) above. We ran the model on two separate samples: students who 
attended elementary schools that ended after 5 th grade, and students who attended elementary 
schools that ended after 6 th grade (see Figure 1). 

These preliminary results are compelling: Students who experience a structural shift 
between 5 th and 6 th grade (blue) exhibit an especially low 6 th grade fall score, which results in what 
looks like an especially large learning loss during the summer. On the other hand, students who 
do not transition between 5 th and 6 th grade do not show any major decrement. At the same time, 
students who transition between 6 th and 7 th grade (orange) experience an unusually large summer 
learning loss in the fall, while the students who transitioned at an earlier grade do not. This 
preliminary work is suggestive that middle school transitions are hard on students — regardless of 
which grade level they experience them in. More nuanced modeling approaches will more formally 
compare the subsequent learning trajectories of pre-treatment similar students in relation to if and 
when they experience middle school transitions. 

Conclusions: 

The phenomenon of middle school transitions is difficult to study in a causal framework, 
either because students in the same schools all make the structural move at the same time (i.e., no 
variation in treatment status), or because the reasons students attend K-4, K-5, K-6, or K-8 
elementary schools may be non-random. Furthermore, it is uncommon for schools to change their 
structure over time, and if they do so, there is probably a reason for doing so that may have its own 
correlated relationship with student achievement. Though we take several approaches to 
addressing these challenges, no one single quasi-experimental approach ensures perfect internal 
validity. Our hope is that, by triangulating several different analytic approaches, we will see a 
preponderance of evidence about whether middle school transitions negatively affect students (and 
perhaps some students more than others). If we do find evidence that middle school transitions 
have lasting, negative effects researchers may wish to explore interventions designed to ease 
these transitions, and policy makers may consider whether certain school grade groupings — like 
K-8 schools, which eliminate one such transition — are more beneficial to students than others. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1. Number of Unique Students in NWEA Dataset for South Carolina, by Grade and 
School Year 


2007-08 

2008-09 

2009-10 

2010-11 

2011-12 


2 

49,230 

49,943 

50,024 

47,669 

44,905 

241,771 

3 

49,671 

52,684 

52,118 

49,390 

45,452 

249,315 

4 

49,206 

50,736 

52,039 

50,231 

45,582 

247,794 

5 

48,086 

49,978 

50,739 

50,308 

46,604 

245,715 

6 

48,515 

48,801 

48,949 

47,358 

43,405 

237,028 

7 

49,760 

49,496 

48,746 

47,288 

43,064 

238,354 

8 

49,885 

49,827 

48,098 

45,779 

43,154 

236,743 


Equation 1. Basic Structure of Two-Level Random Coefficients (Hierarchical) Model 


Level- 1 Model (Repeated Observations of Students (i) over Grade-Sems (i) 
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Vertically Scaled Math RIT Score 
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Figure 1. Estimate Math Achievement From Third Through Seventh Grade for One 
Cohort of South Carolina Students, by Structural Eiementary School Type 
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